U.S. DEPARTMENT OF STATE 
THIRD ANNUAL CONFERENCE ON PROGRAM EVALUATION: 
NEW PARADIGMS FOR EVALUATING 
DIPLOMACY IN THE 2 1ST CENTURY 
June 8-9, 2010 



Programs Track 

PowerPoint presentation from workshop on 
Experience Conducting Evaluation Assessments of Anti— 

Trafficking Programs 

Session transcript : 
http: //www. state . gov/s/d/rm/rls/rm/2010/148062 . htm 



Experiences with Conducting 
Evaluability Assessments of 
Anti-Trafficl<ing Programs 

The State Department's Third Annual 
Conference on Program Evaluation 
June 8-9, 2010 

U.S. Department of State Office to 
Monitor and Combat Trafficking in Persons 

Westat 



Urban Institute 




U.S. Department of State 

Office to Monitor and Combat 
Trafficldng in Persons (G/TIP) 



Conducting Evaluability Assessments of 
Anti-Trajficking Programs 



Jane Nady Sigmon, PhD 

Senior Coordinator for International Programs 




Workshop Goals 

• Provide an overview of the benefits of evaluability 
assessments (EA) and G/TIP's experience 
implementing them 



• Discuss factors to consider when selecting 
programs to undergo an evaluability assessment 

• Describe key steps in planning and conducting 
evaluability assessments 
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Why Evaluability Assessments? 

• Dramatic increase in resources and programming to combat human 
trafficking globally 

• Little empirical data to answer key questions for policy makers and 

donors: 

• Does a program work? 

• Does a program have the intended beneficial effects on the outcomes of 
interest? 

• G/TIP is committed to evaluating program effectiveness and impact 

• First step: assess whether a program is ready for evaluation 

• Well designed, sufficiently developed and documented 

• Are program goals and objectives clear, defined, and realistic? 

• Is program performance data available? 




Two EA Projects Funded in 2009 

• G/TIP published an RFP focused solely on EAs 

• Two organizations selected: Urban Institute and Westat 

• Both organizations given four programs selected by G/TIP 

• Criteria for selection of programs: 

• Project significance-scope of practice and policy likely to be affected 

• Project design clearly linked to objectives and outcomes 

• Type of activity-direct services and prevention programs included 

• Geographic range-6 bilateral projects in 4 countries; 2 regional 
projects 
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Overview of Presentation 



• Background on G/TIP projects involved in the EAs 

• EA research questions 

• EA methodology 

• Technical assistance needs 

• Next steps 



^^a€kgTOTnthm^heT0tHr G/T 
Projects 

• Regions: Near East Asia (NEA), Western Hemisphere 
and the Americas (WHA), and South and Central Asia 
(SCA) 

• Each project has several objectives 

• NEA project (domestic migrant workers): rescue, shelter, 
comprehensive services for trafficked workers, training 
of security staff, hotline, seminars for new migrant 
workers, collaborate with national agencies and NGOs 
to effect policy and procedural changes. 




Examples of Project Activities 

• WHA project: Train project directors, on changing norms 
about the use of sex workers, train-the-trainer workshops, 
peer training; pubUc awareness. 

• SCA project: rescue of forced laborers, aftercare services 
(including 2-year followup), casework assistance, training of 
investigators and lawyers, and structural change in the 
criminal justice system. 

• SCA project: rescue and rehabilitation of labor/sex slaves 
through actions by community committees; mobilize and 
train law enforcement, transport officials, border security; 
comprehensive care and follow-up of survivors; and increase 
capacity of NGOs to rescue and reintegrate victims. 




Questions 

• Project Design 

• To what extent are the project goals reaUstic given the 
activities, inputs, and time frame? 

• To what extent are project performance targets sufficiently 
ambitious yet realistic, given the environment? 



• Project Implementation 

• To what extent is the project implemented as planned? 

• What adaptations are necessary? 

• What are the facilitators and constraints to implementation? 

• To what extent is planning for sustainability part of the 
project design? 



Questions (Continued) 

• Measurement 

• What are the indicators to measure inputs and 
activities? 

• Did the project collect baseline data? 

• To what extent and how are project participants 
tracked? 

• What methods are used to collect, store, and analyze 
project data? 

• To what extent are the data appropriate for measuring 
project impact? 



Methods 



• Letter of introduction sent to projects involved in the 
EAs. 

• Review of project documents: 

• Proposals 

• Other project documents (e.g. case studies, reports of 
training, benchmarks for project activities) 

• Logic models 

• Prepare EA logic models 



Methods (continued) 

• Design data collection tools 

• Conduct key informant telephone interviews with 
NGO headquarters' staff members and in-country 
project directors 

• Plan site visits 

• Conduct Site visits 

• Identify technical assistance needs 

• Provide ongoing technical assistance 



Prepare EA Logic Models 



• Prepare or revise EA logic models 

• To ensure EA evaluators' understanding of project 
design. 

• To compare with project logic models (e.g., inputs versus 
outputs, and outcomes.) 

• To examine the fidelity of the implementation of the 
project. 

• Example of EA logic model (see handout) 




and Next Steps 



^Meeds 



TA is a combination of project requests and EA 
evaluators' recommendations 

• Examples of TA requests from projects 

• Review data collection efforts and provide feedback 

• Recommend an approach to measuring cost/benefit analysis 

• Examples of EA evaluators' recommendations 

• Clarify terminology (e.g., outputs versus outcomes) 

• Advise on ways in which existing data can be used to measure 
service utilization, outcomes, and outputs. 

• Examine project outcomes as a whole 



Identify promising practices 
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URBAN INSTITUTE'S EVALU ABILITY 
ASSESSMENT PROCESS 



Task i: Site Selection and Review Extant Program Materials 
Task 2: Conduct Telephone Interviews and Develop Protocols 
Task 3: Conduct Site Visits to Selected Programs 
Task 4: Prepare Evaluability Assessment Reports 
Task 5: Develop Technical Assistance Strategy 
Task 6: Prepare Final Report 



The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



EVALUABILITY ASSESSMENT SITES 



Regions: South, Central Asia (SCA); Africa (AF) 

AF Project ; Community sensitization; capacity-building; PSC training; mental 
health counseling. 

AF Project ; Create/enhance networking and referral among anti-trafficking 
groups, service providers, law enforcement; review anti-TIP legislation; train 
criminal justice stakeholders; train service providers. 

SCA Project ; Rescue; rehabilitation; reintegration. Rehabilitation (3 centers) 
includes healthcare, psychological counseling, vocational skills training, formal 
education, childcare and parenting skills. 

SCA Project ; Policy advocacy; campaigns and networking; education and 
training, research, documentation and publication, empowerment of survivors. 

The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



WHAT IS NEEDED FOR A FULL IMPACT 

EVALUATION 



• Appropriate sample size and frame 

• Program operates according to logic model 

• Comprehensive data collection tools and processes 

- Informed by program logic model 

- Documents program activities 

- Capacity to collect over time (baseline and follow-up) 

• Assessment of potential threats to validity 

- Co-occurring programming 

- Attrition (client and staff) 

- Inconsistent funding 

- Program buy-in 

- Inconsistent program dosage/activities 

The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



LESSONS LEARNED FROM CONDUCTING 
EVALUABILITY ASSESSMENTS IN SCA and AF 



Establish trust/relationships 
Define research objectives and roles 
Clarify expectations and timeline 
Interpretation/translation 

Adaptability while ensuring adherence to human 
subjects protections 

Anticipate additional costs and time 



The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



INTERIM FINDINGS: RESCUE AND REHABILITATION 

FOR VICTIMS OF SEX TRAFFICKING 



Three holistic victim centers 
Follow-up activities (up to three years) 
Comprehensive database 
Confidential and systematic forms 
Regular program-wide staff meetings 
Staff trainings 
Dedicated staff 



The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



TECHNICAL ASSISTANCE STRATEG 



Assist logic model development 



Improve baseline and follow-up data collection 



Staff training on impact evaluation 



Ensure consistent delivery of services 



The views expressed are those of the authors and should not be attributed to the Urban Institute, its 
trustees, or its funders. 
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OPTIONS FOR DELIVERING T&TA 



• Deliver evaluability assessment and technical 
assistance strategy reports 



• Solicit feedback on reports 



• Communication with sites 



• Longer term technical assistance built into future 
evaluation efforts 

The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



URBAN INSTITUTE CONTACT INFORMATION 



William Adams, badams(a)urban.org 



Meredith Dank, mdank(a)urban . org 



• Colleen Owens, cowens(a)urban.org 



The views expressed are those of the authors and should not be attributed to the Urban Institute, its HI ^''^'^'^ INSTITUTE 
trustees, or its funders. IS Justice Policy Center 



Conclusions 



• Lessons to be learned and next steps 

• EAs will be used to strengthen anti-trafficking efforts and 
provide key stakeholders with useful information 

• Information Dissemination 

• Lessons learned regarding program design, documentation, 
and data collection 

• Promising practices 

• Plans for future evaluations 

• EAs serve as precursors to future impact evaluations 

• Research and evaluation in G/TIP s FY2010 solicitation 

• Impact evaluation projects currently under consideration 



Thank You! 



